1. Field of the Invention
The present invention relates to software development and, more particularly, to performance testing of software application.
2. Description of the Related Art
“On-demand” computing models have become increasingly popular in fields as diverse as finance, media, government, and the like. On-demand computing, referred to as OD computing, is an enterprise model in which computing resources are made available to a user as needed. While the resources can be maintained within the user's own enterprise, the resources also can be provided by service providers outside of the user's enterprise. The OD model allows organizations to deal with fluctuating computing demands in an efficient manner. This is advantageous in that computing resources within an enterprise can vary significantly over time.
The growth of OD computing also has spurred growth in the demand for high performance content management systems. A content management system (CMS) is a software-based system used to manage various forms of electronic content. CMS's are used within business organizations to manage electronic records, whether locally stored or distributed over a Local Area Network (LAN), a Wide Area Network (WAN), or the Internet.
The rise in OD computing, while beneficial from a financial standpoint, introduces a degree of complexity with respect to performance testing. Typically, a firm is engaged to design, size, implement, performance-test, and deploy a CMS. Tasks such as performance testing are performed on a per engagement basis for a specific CMS developed for a particular customer.
Performance testing a CMS is different than performance testing other varieties of computer-based systems. To better illustrate, contrast performance tests relating to a Web server with that of a customized CMS solution. When testing a Web server, the type of request being received is not particularly relevant. That is, whether the request is for audio, video, or text, the Web server performs the same retrieval function. The most significant aspect of the operation is not the nature of the request or the type of document requested, but rather the size of the requested document, i.e. a 10K text document versus a 100K image file.
To test a CMS properly, workloads that accurately reflect customer needs must be designed. For example, an auto insurance company may store thousands of application documents per day, but only search and retrieve documents from a limited collection of claim documents. A governmental organization may perform millions of searches per day, perform millions of document retrieval operations, but perform such operations against a limited data store that includes only several hundred thousand documents. As can be seen, the test scenarios needed to properly test each different type of CMS solution differ significantly from one another.
Performance testing a CMS is different than performance testing other varieties of computer-based systems. To better illustrate, contrast performance test relating to a Web server with that of a customized CMS solution. When testing a Web server, the type of request being received is not particularly relevant. That is, whether the request is for audio, video, or text, the Web server performs the same retrieval function. The most significant aspect of the operation is not the nature of the request or the type of document requested, but rather the size of the requested document, e.g., a 10 K text document versus a 100 K image file.
The development cycle for test cases can be lengthy and cumbersome. One reason is that, to date, there has not been a standard benchmark that defines how different CMS's are to perform throughout a workday, particularly in response to varying system demands. This can lengthen performance test case design time.
Another reason is that in order to reduce the workload description to a test case that can be understood and used by a software tester, a large amount of program code is required in the test execution engine. This code often is inflexible and requires high maintenance. A workload refers to the amount of processing that a computer system has been given to do at a given time. The workload typically specifies CMS functions to be executed as well as some number of users being connected to and interacting with the CMS. A defined workload can be specified as a benchmark when evaluating a computer system in terms of performance.
While test cases are created during development of a CMS, test cases also are used to diagnose problems experienced by customers. Users may experience problems after the CMS has been installed at the customer site. When the customer calls the CMS vendor for support, the vendor must recreate the conditions at the customer premises in an effort to diagnose the problem. This entails creating a workload and reducing the workload to a test case. As noted, this process can be both time consuming and cumbersome.
It would be beneficial to provide users with the ability to create test cases without having to manually code and/or edit each test case.